Add optional telemetry support to the python connector #628

saishreeeee · 2025-07-07T05:15:52Z

What type of PR is this?

Refactor
Feature
Bug Fix
Other

Description

Initial telemetry logs, failure logs, latency logs via telemetry

How is this tested?

Unit tests
E2E Tests
Manually
N/A

Related Tickets & Documents

Design doc

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Advise developers to use Python 3.7, 3.8, or 3.9 until #26 is fixed.

Includes a GitHub Action which checks for a valid sign-off on every proposed commit

* Isolate delay bounding logic * Move error details scope up one-level. * Retry GetOperationStatus if an OSError was raised during execution. Add retry_delay_default to use in this case. * Log when a request is retried due to an OSError. Emit warnings for unexpected OSError codes * Update docstring for make_request * Nit: unit tests show the .warn message is deprecated. DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

* Test with multiple python versions. * Update pyarrow to version 9.0.0 to address issue in relation to python 3.10 & a specific version of numpy being pulled in by pyarrow. Closes #26 Signed-off-by: David Black <dblack@atlassian.com>

@dbaxa

* Update changelog and bump to v2.0.4 * Specifically thank @dbaxa for this change. Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

* Add test: cursors are closed when connection closes Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com>

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com> my [OAuth PR](https://github.com/databricks/databricks-sql-python/runs/8005844758?check_suite_focus=true) is blocked due to dco validation (following error): <img width="1202" alt="Screen Shot 2022-08-25 at 12 05 40 PM" src="https://user-images.githubusercontent.com/22279672/186747897-c9d57586-366f-41f9-aa66-609f2bf3911f.png"> We should try to avoid running dco for internal databricks employees: I am trying to relax the validation based on this guideline: https://github.com/dcoapp/app/blob/main/README.md#skipping-sign-off-for-organization-members and here: https://stackoverflow.com/questions/62969381/is-it-in-line-with-the-dco-that-a-github-sign-off-needs-and-publishes-full-name

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com>

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com> this is undo of #42 till we figure out how to fix dco

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

@sweisdb

This PR: * Adds the foundation for OAuth against Databricks account on AWS with BYOIDP. * It copies one internal module that Steve Weis @sweisdb wrote for Databricks CLI (oauth.py). Once ecosystem-dev team (Serge, Pieter) build a python sdk core we will move this code to their repo as a dependency. * the PR provides authenticators with visitor pattern format for stamping auth-token which later is intended to be moved to the repo owned by Serge @nfx and and Pieter @pietern

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Bump to v2.1.0 and update changelog Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Signed-off-by: Mohit Singla <mohit.singla@databricks.com> Co-authored-by: Moe Derakhshani <moe.derakhshani@databricks.com>

@courtneyholcomb

* Refactor so we can unit test `inject_parameters` * Add unit tests for inject_parameters * Remove inaccurate comment. Per #51, spark sql does not support escaping a single quote with a second single quote. * Closes #51 and adds unit tests plus the integration test provided in #56 Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Co-authored-by: Courtney Holcomb (@courtneyholcomb) Co-authored-by: @mcannamela

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949 Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

github-actions · 2025-07-07T05:27:46Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

This reverts commit 8c0f474. Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-07T05:29:38Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-07T05:31:47Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-07T05:39:09Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

src/databricks/sql/exc.py

src/databricks/sql/telemetry/latency_logger.py

vikrantpuppala · 2025-07-07T06:50:51Z

src/databricks/sql/telemetry/models/enums.py

+class StatementType(Enum):
+    NONE = "none"
+    QUERY = "query"
+    SQL = "sql"


query type is used when sql command is expected to return results, in jdbc it makes sense due to the nature of interface, but do we need both query and sql in python connector?

the execute and execute_async functions have StatementType set as query and functions like _handle_staging_get, _handle_staging_put and _handle_staging_remove have it set as sql.

vikrantpuppala · 2025-07-07T06:51:29Z

src/databricks/sql/telemetry/models/enums.py

+    CLIENT_CERT = "CLIENT_CERT"  # ssl certificate authentication
+    PAT = "PAT"  # Personal Access Token authentication
+    DATABRICKS_OAUTH = "DATABRICKS_OAUTH"  # Databricks-managed OAuth flow
+    EXTERNAL_AUTH = "EXTERNAL_AUTH"  # External identity provider (AWS, Azure, etc.)


I think @jprakash-db is adding some enums in his latest PR #621, is there scope to re-use?

src/databricks/sql/telemetry/telemetry_client.py

…or/ResultSet, shifted log_latency decorator to fetchall Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-08T06:32:04Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-08T06:33:54Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-08T06:36:35Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

github-actions · 2025-07-08T06:57:39Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Jesse and others added 30 commits June 28, 2022 10:53

Add CODEOWNERS (#10)

71defd5

Reformat changelog (#11)

187cbe8

Add e2e tests (#12)

e5a5001

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Indicate that Python 3.10 is not supported (#27)

86969b1

Advise developers to use Python 3.7, 3.8, or 3.9 until #26 is fixed.

Add Developer Certificate of Origin requirement (#13)

abaa45a

Includes a GitHub Action which checks for a valid sign-off on every proposed commit

Bump to v2.0.3 (#28)

10d765b

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Bump version to 2.0.4-dev (#29)

34397db

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

[PECO-197] Support Python 3.10 (#31)

5fd729a

* Test with multiple python versions. * Update pyarrow to version 9.0.0 to address issue in relation to python 3.10 & a specific version of numpy being pulled in by pyarrow. Closes #26 Signed-off-by: David Black <dblack@atlassian.com>

Update changelog and bump to v2.0.4 (#34)

5612e5f

* Update changelog and bump to v2.0.4 * Specifically thank @dbaxa for this change. Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Bump to 2.0.5-dev on main (#35)

f1241ba

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

On Pypi, display the "Project Links" sidebar. (#36)

6bafb9d

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

[ES-402013] Close cursors before closing connection (#38)

afcb0f0

* Add test: cursors are closed when connection closes Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

Bump version to 2.0.5 and improve CHANGELOG (#40)

af945aa

Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev>

fix dco issue

441a6ae

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com>

fix dco issue

29fe6b4

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com>

dco tunning

cf3130e

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com>

dco tunning

4387f93

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com>

Merge pull request #43 from moderakh/dco-tunning

285e516

Signed-off-by: Moe Derakhshani <moe.derakhshani@databricks.com> this is undo of #42 till we figure out how to fix dco

Github workflows: run checks on pull requests from forks (#47)

ea0f076

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Automate deploys to Pypi (#48)

e39d294

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

[PECO-205] Add functional examples (#52)

1ea2fe0

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Bump version to 2.1.0 (#54)

3638fa2

Bump to v2.1.0 and update changelog Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

[SC-110400] Enabling compression in Python SQL Connector (#49)

1a4cf4b

Signed-off-by: Mohit Singla <mohit.singla@databricks.com> Co-authored-by: Moe Derakhshani <moe.derakhshani@databricks.com>

Bump thrift dependency to 0.16.0 (#65)

3d3c692

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949 Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Bump version to 2.2.0 (#66)

5cbfcac

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

Support Python 3.11 (#60)

c6e573c

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>

saishreeeee had a problem deploying to azure-prod July 7, 2025 05:27 — with GitHub Actions Failure

Revert "Revert "Merge branch 'main' into telemetry""

13ebfb4

This reverts commit 8c0f474. Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee had a problem deploying to azure-prod July 7, 2025 05:29 — with GitHub Actions Failure

workflows

79db09f

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee had a problem deploying to azure-prod July 7, 2025 05:31 — with GitHub Actions Failure

-

5005561

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee temporarily deployed to azure-prod July 7, 2025 05:39 — with GitHub Actions Inactive

saishreeeee marked this pull request as ready for review July 7, 2025 05:40

saishreeeee requested review from jprakash-db and vikrantpuppala July 7, 2025 05:41

vikrantpuppala changed the title ~~Telemetry~~ Add optional telemetry support to the python connector Jul 7, 2025

vikrantpuppala reviewed Jul 7, 2025

View reviewed changes

changed enums to follow proto, get_extractor returns None if not Curs…

96813ad

…or/ResultSet, shifted log_latency decorator to fetchall Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee temporarily deployed to azure-prod July 8, 2025 06:31 — with GitHub Actions Inactive

formatting

846e701

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee temporarily deployed to azure-prod July 8, 2025 06:33 — with GitHub Actions Inactive

auth mech test fix

d939fe3

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee temporarily deployed to azure-prod July 8, 2025 06:36 — with GitHub Actions Inactive

saishreeeee requested a review from vikrantpuppala July 8, 2025 06:55

import logging

cac9c7a

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

saishreeeee deployed to azure-prod July 8, 2025 06:57 — with GitHub Actions Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add optional telemetry support to the python connector #628

Add optional telemetry support to the python connector #628

saishreeeee commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vikrantpuppala Jul 7, 2025

Uh oh!

saishreeeee Jul 8, 2025

Uh oh!

vikrantpuppala Jul 7, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Add optional telemetry support to the python connector #628

Are you sure you want to change the base?

Add optional telemetry support to the python connector #628

Conversation

saishreeeee commented Jul 7, 2025

What type of PR is this?

Description

How is this tested?

Related Tickets & Documents

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vikrantpuppala Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

saishreeeee Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

vikrantpuppala Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!